8.2.1 Check out the hyper-parameters for each algo

?h2o.glm 
?h2o.randomForest
?h2o.gbm
?h2o.deeplearning
?h2o.xgboost

8.2.2 Train a xgboost model with manual settings

model_gbm_DPOL_m <- h2o.gbm(x = features, 
                            y = target_DPOL, 
                            training_frame = h_train,
                            nfolds = 5,
                            seed = n_seed,
                                
                            # Manual Settings based on experience
                            learn_rate = 0.1,       # use a lower rate (more conservative)
                            ntrees = 120,           # use more trees (due to lower learn_rate)
                            sample_rate = 0.7,      # use random n% of samples for each tree  
                            col_sample_rate = 0.7)  # use random n% of features for each tree

8.2.3 Comparison (RMSE: Lower = Better)

# Create a table to compare RMSE from different models
d_eval <- data.frame(model = c("GBM: Gradient Boosting Model (Baseline)",
                               "GBM: Gradient Boosting Model (Manual Settings)"),
                     stringsAsFactors = FALSE)
d_eval$RMSE_cv <- NA
d_eval$RMSE_test <- NA

# Store RMSE values
d_eval[1, ]$RMSE_cv <- model_gbm_DPOL@model$cross_validation_metrics@metrics$RMSE
d_eval[2, ]$RMSE_cv <- model_gbm_DPOL_m@model$cross_validation_metrics@metrics$RMSE
d_eval[1, ]$RMSE_test <- h2o.rmse(h2o.performance(model_gbm_DPOL, newdata = h_test))
d_eval[2, ]$RMSE_test <- h2o.rmse(h2o.performance(model_gbm_DPOL_m, newdata = h_test))

# Show Comparison (RMSE: Lower = Better)
kable(d_eval)

8.3.1 Leaderboard

# Show the table
kable(as.data.frame(automl_DPOL@leaderboard))

8.3.2 Best Model (Leader)

# Show the best model
automl_DPOL@leader

Model Details:
==============

H2ORegressionModel: gbm
Model ID:  GBM_grid__1_AutoML_20210517_094305_model_7 
Model Summary: 
  number_of_trees number_of_internal_trees model_size_in_bytes min_depth
1              43                       43               10607         4
  max_depth mean_depth min_leaves max_leaves mean_leaves
1         4    4.00000          9         16    13.79070


H2ORegressionMetrics: gbm
** Reported on training data. **

MSE:  742.9795
RMSE:  27.25765
MAE:  7.378285
RMSLE:  NaN
Mean Residual Deviance :  742.9795



H2ORegressionMetrics: gbm
** Reported on cross-validation data. **
** 5-fold cross-validation on training data (Metrics computed for combined holdout predictions) **

MSE:  4022.796
RMSE:  63.42551
MAE:  15.04154
RMSLE:  NaN
Mean Residual Deviance :  4022.796


Cross-Validation Metrics Summary: 
                            mean         sd cv_1_valid cv_2_valid cv_3_valid
mae                    15.045994  3.9332395   12.12232   11.87795  17.959179
mean_residual_deviance 4025.1875  2845.8782  2456.1912  2354.3364   8599.125
mse                    4025.1875  2845.8782  2456.1912  2354.3364   8599.125
r2                     0.5540781 0.14167649  0.5431397  0.5116641 0.44749096
residual_deviance      4025.1875  2845.8782  2456.1912  2354.3364   8599.125
rmse                     60.5994  21.002983  49.559975  48.521503   92.73147
rmsle                        NaN        0.0        NaN        NaN        NaN
                       cv_4_valid cv_5_valid
mae                     20.492651  12.777869
mean_residual_deviance   4990.901  1725.3843
mse                      4990.901  1725.3843
r2                     0.46939796  0.7986977
residual_deviance        4990.901  1725.3843
rmse                     70.64631  41.537746
rmsle                         NaN        NaN

8.3.3 Comparison (RMSE: Lower = Better)

d_eval_tmp <- data.frame(model = "Best Model from H2O AutoML",
                         RMSE_cv = automl_DPOL@leader@model$cross_validation_metrics@metrics$RMSE,
                         RMSE_test = h2o.rmse(h2o.performance(automl_DPOL@leader, newdata = h_test)))
d_eval <- rbind(d_eval, d_eval_tmp)

# Show the table
kable(d_eval)

8.5.1 Global Explanations

# Show global explanations for best model from AutoML
h2o.explain(automl_DPOL@leader, newdata = h_test)

Residual Analysis
=================

> Residual Analysis plots the fitted values vs residuals on a test dataset. Ideally, residuals should be randomly distributed. Patterns in this plot can indicate potential problems with the model selection, e.g., using simpler model than necessary, not accounting for heteroscedasticity, autocorrelation, etc. Note that if you see "striped" lines of residuals, that is an artifact of having an integer valued (vs a real valued) response variable.

Variable Importance
===================

> The variable importance plot shows the relative importance of the most important variables in the model.

SHAP Summary
============

> SHAP summary plot shows the contribution of the features for each instance (row of data). The sum of the feature contributions and the bias term is equal to the raw prediction of the model, i.e., prediction before applying inverse link function.

Partial Dependence Plots
========================

> Partial dependence plot (PDP) gives a graphical depiction of the marginal effect of a variable on the response. The effect of a variable is measured in change in the mean response. PDP assumes independence between the feature for which is the PDP computed and the rest.

Individual Conditional Expectations
===================================

> An Individual Conditional Expectation (ICE) plot gives a graphical depiction of the marginal effect of a variable on the response. ICE plots are similar to partial dependence plots (PDP); PDP shows the average effect of a feature while ICE plot shows the effect for a single instance. This function will plot the effect for each decile. In contrast to the PDP, ICE plots can provide more insight, especially when there is stronger feature interaction.

8.5.2 Local Explanations

# Show local explanations
h2o.explain_row(automl_DPOL@leader, newdata = h_test, row_index = 1)

SHAP explanation
================

> SHAP explanation shows contribution of features for a given instance. The sum of the feature contributions and the bias term is equal to the raw prediction of the model, i.e., prediction before applying inverse link function. H2O implements TreeSHAP which when the features are correlated, can increase contribution of a feature that had no influence on the prediction.

Individual Conditional Expectations
===================================

> Individual conditional expectations (ICE) plot gives a graphical depiction of the marginal effect of a variable on the response for a given row. ICE plot is similar to partial dependence plot (PDP), PDP shows the average effect of a feature while ICE plot shows the effect for a single instance.

8.5.3 Local Contributions

# Make Predictions
predictions <- h2o.predict(automl_DPOL@leader, newdata = h_test)

# Show the table
kable(head(predictions, 5))

# Calculate feature contributions for each sample
contributions <- h2o.predict_contributions(automl_DPOL@leader, newdata = h_test)

# Show the table
kable(head(contributions, 5))